Apparatus and method for configuring screen for video call using facial expression

ABSTRACT

Apparatus and method for configuring a screen for a video call using a facial expression by recognizing a face from an image, calculating facial expression information for an expression of the recognized face, and determining whether there is a change in expression of the recognized face by comparing the calculated facial expression information with reference expression information preset to determine a change in expression of the face. If there is a change in expression of the recognized face, the apparatus and method selects a video image corresponding to the changed expression in the video call screen, and reconfigures the video call screen using the selected video image, making it possible for a user to conveniently select an image of the interested person without taking extensive action, and preventing a wrong image from being selected due to the unintended user facial movement.

PRIORITY

This application is a Continuation Application of U.S. application Ser.No. 13/293,720, filed at the U.S. Patent and Trademark Office on Nov.10, 2011, now U.S. Pat. No. 8,810,624, issued on Aug. 19, 2014, whichclaims priority under 35 10 U.S.C. §119(a) to a Korean PatentApplication filed in the Korean Intellectual Property

Office on Nov. 10, 2010 and assigned Serial No. 10-2010-0111791, theentire disclosure of which is incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to a screen configuringapparatus and method, and more particularly, to an apparatus and methodfor configuring a screen for a video call by selecting an image of aninterested user among multiple users.

2. Description of the Related Art

In a video call, a picture of a caller is taken using a camera,displayed on a screen, and images of the persons with whom the callerwants to have a telephone conversation are displayed in a specificlocation of the screen, for a video call.

A multipoint video call (or video conference call) technique, whichallows a user to have a video call with multiple persons on a mobileterminal, automatically identifies a speaking party by lip movementrecognition, and displays an image of the speaker at the center of thescreen, making it possible to talk with multiple persons.

In a multipoint video call apparatus, a display includes a main screenhaving the largest area on a video call screen, and at least one subscreen.

However, the conventional multipoint video call technique maymalfunction when several users move their lips at the same time.

Additionally, it is difficult for a user to select an image of anotheruser other than the speaker, when a user has an interest in having aconversation with the other user.

SUMMARY OF THE INVENTION

Accordingly, the present invention has been made to solve theabove-mentioned problems occurring in the prior art, and the presentinvention provides a video call apparatus and method for estimating afacial expression during a video call with multiple users, selecting animage of an interested person, and allowing a user to have a video callwith the selected interested person.

According to one aspect of the present invention, there is provided anapparatus for configuring a screen for a video call using a facialexpression which includes a facial expression information calculator forrecognizing a face from an image, and calculating facial expressioninformation for an expression of the recognized face; a facialexpression determiner for determining whether there is a change inexpression of the recognized face by comparing the calculated facialexpression information with reference expression information preset todetermine a change in expression of the face; a screen configurer forconfiguring a video call screen including multiple video images receivedfor the video call; and an image selector for selecting a video imagecorresponding to the changed expression in the video call screen ifthere is a change in expression. The screen configurer may reconfigurethe video call screen using the selected video image.

According to another aspect of the present invention, there is provideda method for configuring a screen for a video call using a facialexpression by configuring a video call screen including multiple videoimages received for the video call; recognizing a face from an image,and calculating facial expression information for an expression of therecognized face; determining whether there is a change in expression ofthe recognized face by comparing the calculated facial expressioninformation with reference expression information preset to determine achange in expression of the face; selecting a video image correspondingto the changed expression in the video call screen if there is a changein expression; and reconfiguring the video call screen using theselected video image.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages of variousembodiments of the present invention will be more apparent from thefollowing description taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 is a diagram illustrating a structure of a screen configuringapparatus according to an embodiment of the present invention;

FIG. 2 is a flowchart illustrating a process of extracting referenceexpression information used to estimate changes in facial expression ina screen configuring apparatus according to an embodiment of the presentinvention;

FIG. 3 is a diagram illustrating images obtained in a process ofextracting reference expression information according to an embodimentof the present invention;

FIG. 4 is a flowchart illustrating a process of reconfiguring a videocall screen corresponding to changes in facial expression during a videocall in a screen configuring apparatus according to an embodiment of thepresent invention; and

FIGS. 5 to 7 are diagrams illustrating images obtained in a process ofconfiguring a video call screen according to an embodiment of thepresent invention.

Throughout the drawings, the same drawing reference numerals will beused to refer to the same elements, features and structures.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE PRESENT INVENTION

Various embodiments of the present invention will be described in detailwith reference to the accompanying drawings. In the followingdescription, specific details such as detailed configuration andcomponents are merely provided to assist the overall understanding ofvarious embodiments of the present invention. Therefore, it will beapparent to a person having ordinary skill in the art of the presentinvention that various changes and modifications of the embodimentsdescribed herein can be made without departing from the scope and spiritof the invention. In addition, descriptions of well-known functions andconstructions are omitted for clarity and conciseness.

FIG. 1 is a diagram illustrating a structure of a screen configuringapparatus according to an embodiment of the present invention.

Referring to FIG. 1, the screen configuring apparatus includes a facialexpression information calculator 100, a facial expression determiner110, an image selector 120, and a screen configurer 130.

The facial expression information calculator 100 calculates facialexpression information within a frame of an input image received from acamera during a video call, or of input images received outside thevideo call.

The facial expression information calculator 100 presets referenceexpression information that is used to determine changes in facialexpression during a video call from an input image received from thecamera before the video call.

The facial expression information calculator 100 includes a facerecognizer 101, a facial feature extractor 102, and a face anglecalculator 103.

The face recognizer 101 uses a general face recognition technique inrecognizing a face area in an input image, for example, recognizing anarea corresponding to a preset facial skin color in an input image, as aface area.

The facial feature extractor 102 extracts facial features in therecognized face area. For the extraction of the facial features, ageneral facial feature extraction technique is used. The facial featuresas used herein may refer to facial feature components such as eyes,nose, mouth and chin.

The face angle calculator 103 calculates a reference face angle based onthe extracted facial features. Specifically, the face angle calculator103 draws polygonal sides by connecting the calculated facial features,and calculates an angle of the recognized face based on the drawnpolygonal sides. For the calculation of the face angle, a general faceangle calculation technique is used.

When a video call begins, the screen configurer 130 configures a videocall screen for the video call, using at least one input image receivedduring the video call and a user image received from a camera. The atleast one input image is defined as at least one sub image, and the userimage received from a camera is defined as a main image.

That is, the face configurer 130 displays a main image in an area with apreset size on the video call screen, and displays at least one subimage in the remaining area except for the area where the main image isdisplayed. The screen configurer 130 sets a size of the area where themain image is displayed on the video call screen, to be greater than asize of the area where the at least one sub image is displayed.

The facial expression determiner 110 determines whether there is achange in facial expression by comparing facial expression informationin the main image, calculated by the facial expression informationcalculator 100 during a video call, with preset reference expressioninformation.

Specifically, the facial expression determiner 110 determines whetherthere is a change in face angle by comparing the face angle in the mainimage calculated by the face angle calculator 103 with a presetreference face angle.

If there is a change in facial expression information, the imageselector 120 selects a sub image corresponding to the changed facialexpression from among multiple sub images located on the video callscreen.

That is, if a difference between the calculated face angle in the mainimage and the preset reference face angle is greater than or equal to apreset value, the image selector 120 estimates a face directioncorresponding to the face angle in the main image, and selects a subimage corresponding to the estimated face direction in a face area ofthe main image on the video call screen.

If a change in facial expression is continuously recognized for a presettime, the screen configurer 130 reconfigures a video call screencorresponding to the changed facial expression using the sub imageselected by the image selector 120, and displays the reconfigured videocall screen.

Specifically, if the face direction estimated by the image selector 120is continuously recognized for a preset time, the screen configurer 130switches between a screen of the main image and a screen of the selectedsub image on the video call screen.

As such, the screen configuring apparatus estimates a facial expressionof a user and selects an image of an interested person on the video callscreen, making it possible for the user to conveniently select an imageof the interested person without taking extensive action.

FIG. 2 is a diagram illustrating a process of setting referenceexpression information in a screen configuring apparatus according to anembodiment of the present invention.

Referring to FIG. 2, upon receiving an image from a camera in step 200,the facial expression information calculator 100 recognizes a face inthe received image in step 210. As described above, for the recognitionof a face in an image, a general face recognition technique is used, anda technique of learning a skin color and recognizing an areacorresponding to the learned skin color as a face area may also be used.For example, with reference to FIG. 3, the facial expression informationcalculator 100 recognizes a face area 301 in an input image 300.

In step 220, the facial expression information calculator 100 extractsfacial features in the recognized face. As represented by referencenumeral 310, the facial expression information calculator 100 extractsfacial features at the locations corresponding to eyes, nose, mouth andchin in the face area.

In step 230, the facial expression information calculator 100 calculatesa face angle of the recognized face based on the extracted facialfeatures. For example, the facial expression information calculator 100calculates a face angle 321 in an image 320 using an area of a polygonby connecting the facial features, and then ends setting the referenceexpression information.

As such, the screen configuring apparatus may recognize changes infacial expression in an image received during a video call andreconfigure a video call screen corresponding to the changed facialexpression.

FIG. 4 is a diagram illustrating a process of reconfiguring a video callscreen corresponding to changes in facial expression during a video callin a screen configuring apparatus according to an embodiment of thepresent invention.

According to an embodiment of the present invention, a user imagereceived from a camera is defined as a main image, and at least oneinput image received from outside is defined as at least one sub image.An embodiment of the present invention will be described with referenceto FIGS. 5 to 7.

FIGS. 5 to 7 are diagrams illustrating images obtained in a process ofconfiguring a video call screen according to an embodiment of thepresent invention.

Referring to FIG. 4, if a video call begins in step 400, the screen configurer 130 configures and displays a video call screen including ascreen of a main image and a screen of at least one sub image in step401.

The screen configurer 130 displays a main image in an area with a presetsize on the video call screen, and displays at least one sub image inthe remaining area except for the area where the main image isdisplayed. The screen configurer 130 sets a size of the area where themain image is displayed on the video call screen, to be greater than asize of the area where the at least one sub image is displayed.

The displayed video call screen may be as illustrated in FIG. 5.

In step 402, the facial expression information calculator 100 recognizesa face in the main image, calculates facial features of the recognizedface, and calculates a face angle based on the calculated facialfeatures.

Specifically, the face recognizer 101 recognizes a face area in the mainimage using the general face recognition technique, for example, byrecognizing an area corresponding to a preset facial skin color in aninput image, as a face area.

Thereafter, the facial feature extractor 102 extracts facial features inthe recognized face area, and the face angle calculator 103 calculates areference face angle based on the extracted facial features.

In step 403, the facial expression determiner 110 compares the faceangle in the main image calculated by the facial expression informationcalculator 100, with a preset reference face angle.

In step 404, the facial expression determiner 110 determines whetherthere is a change in face angle. If there is a change in face angle, theimage selector 120 proceeds to step 405. Otherwise, the screenconfigurer 130 continuously displays the video call screen in step 401.

In step 405, the image selector 120 selects a sub image that is locatedon the video call screen to correspond to the face angle of the mainimage.

That is, if a difference between the calculated face angle in the mainimage and the preset reference face angle is greater than or equal to apreset value, the image selector 120 estimates a face directioncorresponding to the face angle in the main image, and selects a subimage corresponding to the estimated face direction in a face area ofthe main image on the video call screen.

Referring to FIGS. 5 and 6, the image selector 120 estimates a facedirection 502 corresponding to a face angle of a main image 500, andselects a sub image 501 corresponding to the estimated face direction502 in the face area.

As illustrated in FIG. 5, the screen configurer 130 may further display,on the video call screen, face direction arrow icons for allowing theuser to recognize face directions corresponding to face angles. Theseface direction arrow icons may be displayed to overlap the screen of themain image.

To emphasize that the selected sub image is a selected image, the screenconfigurer 130 may display the edge of the selected sub image to bebold, or may display the selected sub image to be greater in size thanother sub images.

In step 406, the screen configurer 130 determines whether a change inface angle is continuously recognized for a preset time. If the changein face angle is continuously recognized, the screen configurer 130proceeds to step 407. Otherwise, the screen configurer 130 continuouslydisplays the video call screen in step 401.

The reason why the screen configurer 130 determines whether a change inface angle continues for a preset time is to prevent a wrong sub imagefrom being selected due to the unintended user facial movement.

In step 407, the screen configurer 130 reconfigures a video call screencorresponding to the changed facial expression using the sub imageselected by the image selector 120, and displays the reconfigured videocall screen. For example, as illustrated in FIG. 6, the screenconfigurer 130 reconfigures the video call screen such that the screenof the main image and the screen of the selected sub image are switchedin terms of the location, thereby displaying the selected sub image inthe area of the main image.

As illustrated in FIG. 7, the screen configurer 130 may to place thescreen of the sub image in the area of the main image by switchingbetween a screen 700 of the main image and a screen 701 of the sub imageon the video call screen.

In step 408, the screen configurer 130 determines if the video call hasbeen completed. If the video call has been completed, the screen configurer 130 ends the video call operation. Otherwise, the screenconfigurer 130 returns to step 401 and repeats its succeeding steps.

According to embodiments of the present invention, the user's facialexpression is estimated and an image of the user's interested person onthe video call screen is selected, thereby allowing the user toconveniently select an image of the interested person without takingextensive action. The embodiments of the present invention may set atime margin for selecting an image of an interested person, to prevent awrong image from being selected due to unintended user facial movement.

In addition, the embodiments of the present invention provide accurate,intuitive and convenient functions according to the display screen ofthe video call apparatus, thereby increasing user convenience.

While the invention has been shown and described with reference tovarious embodiments thereof, it will be understood by those skilled inthe art that various changes in form and detail may be made thereinwithout departing from the spirit and scope of the invention as definedby the appended claims.

What is claimed is:
 1. An apparatus for configuring a screen for a videocall using a facial expression, comprising: a facial expressioninformation calculator for recognizing a face from an image, andcalculating facial expression information for an expression of therecognized face; a facial expression determiner for determining whetherthere is a change in expression of the recognized face by comparing thecalculated facial expression information with reference expressioninformation preset to determine a change in expression of the face; ascreen configurer for configuring a video call screen including multiplevideo images received for the video call; and an image selector forselecting a video image corresponding to the changed expression in thevideo call screen if there is a change in expression, wherein the screenconfigurer reconfigures the video call screen using the selected videoimage.
 2. The apparatus of claim 1, wherein the video call screenincludes a screen of a main image received from a camera and a screen ofat least one sub image received outside of the video call.
 3. Theapparatus of claim 2, wherein the facial expression informationcalculator recognizes a face in the main image, extracts facial featuresof the recognized face, and calculates a face angle of the recognizedface based on the extracted facial features.
 4. The apparatus of claim3, wherein the facial expression information calculator calculates aface angle from an image received before the video call, and sets thecalculated face angle as a reference face angle.
 5. The apparatus ofclaim 4, wherein the facial expression determiner determines whether adifference between the calculated face angle and the reference faceangle is greater than or equal to a preset threshold by comparing thecalculated face angle with the reference face angle.
 6. The apparatus ofclaim 5, wherein the image selector estimates a face directioncorresponding to the calculated face angle if the difference between thecalculated face angle and the reference face angle is greater than orequal to the threshold and selects a sub image corresponding to theestimated face direction in a face area of the main image on the videocall screen.
 7. The apparatus of claim 6, wherein the screen configurerreconfigures the video call screen by switching between the screen ofthe main image and the screen of the selected sub image on the videocall screen.
 8. A method for configuring a screen for a video call usinga facial expression, comprising: configuring a video call screenincluding multiple video images received for the video call; recognizinga face from an image, and calculating facial expression information foran expression of the recognized face; determining whether there is achange in expression of the recognized face by comparing the calculatedfacial expression information with reference expression informationpreset to determine a change in expression of the face; selecting avideo image corresponding to the changed expression in the video callscreen if there is a change in expression; and reconfiguring the videocall screen using the selected video image.
 9. The method of claim 8,wherein the video call screen includes a screen of a main image receivedfrom a camera and a screen of at least one sub image received outside ofthe video call.
 10. The method of claim 9, wherein calculating facialexpression information further comprises: recognizing a face in the mainimage; extracting facial features of the recognized face; andcalculating a face angle of the recognized face based on the extractedfacial features.
 11. The method of claim 10, further comprisingcalculating a face angle from an image received before the video call,and setting the calculated face angle as a reference face angle.
 12. Themethod of claim 11, wherein determining whether there is a change inexpression of the recognized face further comprises determining whethera difference between the calculated face angle and the reference faceangle is greater than or equal to a preset threshold by comparing thecalculated face angle with the reference face angle.
 13. The method ofclaim 12, wherein selecting a video image corresponding to the changedexpression comprises: estimating a face direction corresponding to thecalculated face angle if the difference between the calculated faceangle and the reference face angle is the threshold value or above; andselecting a sub image corresponding to the estimated face direction in aface area of the main image on the video call screen.
 14. The method ofclaim 13, wherein reconfiguring the video call screen comprisesswitching between the screen of the main image and the screen of theselected sub image on the video call screen.